WORLD INTELLECTUAL PROPERTY ORGANIZATION 
Iniemational Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification ^ : 
G06K 9/22, H04N 1/047, 17107 



Al 



(11) International Publication Number: WO 98/20446 

(43) International Publication Date: 14 May 1998 (14.05.98) 



(21) International Application Number: PCT/SE97/0i827 

(22) International Filing Date: 3 November 1997 (03. 11. 97) 



(30) Priority Data: 
9604008-4 



1 November 1996 (01.11. 96) 



SE 



(71) Applicant {for all designated States except US): C TECH- 

NOLOGIES AB [SE/SE]; Forskningsbyn Ideon. S-223 70 
Lund (SE). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): fAhRAEUS, Christer 
[SE/SE]; Frcdsgatan 1, III, S-222 20 Lund (SE). NILSSON, 
Jan [SE/SE]; Stora SOdergatan 11, S-222 23 Lund (SE). 
SODERLUND, Patrik [SE/SE]; Stora RSby, MAnstorp, 
S-224 78 Lund (SE). 



(81) Designated States: AL, AM, AT, AT (Utility model), AU, AZ, 
BA, BB. BG, BR. BY, CA, CH, CN. CU, C2, CZ (Utility 
model), DE, DE (Utility model), DK, DK (Utility model), 
EE, EE (Utility model), ES, FI, Fl (Utility model). GB, GE, 
GH. HU. ID, IL, IS, JP, KE, KG, KP, KR. KZ, LC, LK. 
LR, LS, LT, LU, LV, MD, MG, MK. MN, MW, MX, NO, 
NZ, PL. PT, RO, RU. SD. SE. SG, SI. SK, SK (Utility 
model). SL. TJ. TM. TR. TT, UA, UG. US, UZ, VN, YU, 
ZW. ARIPO patent (GH, KE, LS, MW, SD, SZ. UG, ZW), 
Eurasian patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), 
European patent (AT, BE, CH, DE, DK, ES. FI, FR, GB, 
GR, IE, IT. LU. MC. NL, PT. SE). OAPI patent (BF, BJ, 
CF, CG, CI, CM, GA, GN, ML, MR, NE, SN. TD, TG). 



Published 

With international search report. 

In English translation (filed in Swedish). 



(74) Agent: AWAPATENT AB; P.O. Box 5117, S-200 71 Malm6 
(SE). 



(54) Title: RECORDING METHOD AND APPARATUS 




(57) Abstract 



In a method for recording a selected sequence of characters with the aid of a pen, the latter is passed over the sequence of characters 
and a plurality of images, whose contents partially overlap, are recorded simultaneously with the aid of a light-sensitive sensor (8) with 
a two-dimensional sensor surface. The recorded images arc put together, horizontally as well as vertically, into an image. Subsequently, 
the characters in the composite image are identified and stored in the pen with the aid of a predetermined character code format. The 
putting-together, the character identification, and the storing are carried out with the aid of a signal processing unit (20-24). By the method 
and the pen compensation is obtained for different speeds of movement and different angles of rotation of the pen. 



FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT. 



AL Albania 

AM Amicnia 

AT Austria 

AU Australia 

AZ Azerbaijan 

BA Bosnia and Herzegovina 

BB Barbados 

BE Belgium 

BF Burkina Faso 

EG Bulgaria 

BJ Benin 

BR Brazil 

BY Belarus 

CA Canada 

CF Central African Republic 

CG Congo 

CH Switzerland 

CI C6tc d'lvoire 

CM Cameroon 

CN China 

CU Cuba 

CZ Czech Republic 

D£ Germany 

DK Denmark 

EE Estonia 



ES 


Spain 


LS 


Lesotho 


SI 


FI 


Finland 


LT 


Lithuania 


SK 


FR 


France 


LU 


Luxembourg 


SN 


GA 


Gabon 


LV 


Latvia 


sz 


GB 


United Kingdom 


MC 


Monaco 


TD 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


GH 


Ghana 


MG 


Madagascar 


TJ 


GN 


Guinea 


MR 


The former Yugoslav 


TM 


GR 


Greece 




Republic of Macedonia 


TR 


HU 


Hungary 


ML 


Mali 


TT 


IE 


Ireland 


MN 


Mongolia 


UA 


IL 


Israel 


MR 


Mauritania 


UG 


IS 


Iceland 


MW 


Malawi 


US 


IT 


Italy 


MX 


Mexico 


UZ 


JP 


Japan 


NE 


Niger 


VN 


KE 


Kenya 


NL 


Netherlands 


YU 


KG 


Kyrgyzstan 


NO 


Norway 


ZW 


KP 


Democratic People's 


NZ 


New Zealand 






Republic of Korea 


PL 


Poland 




KR 


Republic of Korea 


PT 


Portugal 




KZ 


Kazakstan 


RO 


Romania 




LC 


Saint Lucia 


RU 


Russian Federation 




U 


Liechtenstein 


SD 


Sudan 




LK 


Sri Lanka 


SE 


Sweden 




LR 


Liberia 


SG 


Sjng^}ore 





Slovenia 

Slovakia 

Senegal 

Swaziland 

Chad 

Togo 

Tajikistan 

Turkmenistan 

Turkey 

Trinidad and Tobago 

Ukraine 

Uganda 

United States of America 

Uzbekistan 

Viet Nam 

Yugoslavia 

Zimbabwe 



wo 98/20446 



PCT/SE97/01827 



1 

RECORDING METHOD AND APPARATUS 

Field of the Invention 

The present invention relates to a pen for record- 
ing a selected sequence of characters according to the 
preamble of claim 1, as well as a method for effecting 
5 such a recording according to the preamble of claim 7. 
Background of the Invention 

A reader who reads a text often wishes to mark the 
most important sections of the text in order to be able 
to easily find them when next reading through the text, 

10 thereby avoiding having to read irrelevant parts of the 
text. It is known to use so-called highlighter pens for 
this purpose. 

A reader may also need to put together selected 
parts of a text into a new document, which can be edit- 

15 ed in a word-processing program in a computer. A known 

method for feeding text into a computer is to use a scan- 
ner. There are stationary scanners and portable scanners, 
A stationary scanner is used to scan whole pages of text, 
the scanner being passed over the text automatically at a 

20 constant speed. This type of scanner is not suitable for 
scanning selected parts of a text on a page. A portable 
scanner, on the other hand, can be used for this purpose. 

US 5,301,243 discloses a handheld scanner for read- 
ing characters from a character string on a substrate. 

25 The scanner, which is passed over the characters which 
are to be read in contact with the substrate, has an 
optical system which "sees" a small part of the sub- 
strate. The optical system comprises a CCD type line 
sensor, which has a plurality of light-sensitive ele- 

30 ments arranged in a row. When the scanner is passed over 
the characters on the substrate, a sequence of vertical 
"slices" of the characters and of the spaces between them 
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is recorded. The "slices" are stored in the scanner as a 
digital bitmap. Subsequently, OCR software (OCR = Optical 
Character Recognition) is used to identify the scanned 
characters and to store them in character-coded format, 
5 e.g. with the aid of ASCII code, in a memory. The charac- 
ter recognition can be performed either in the scanner or 
in an external computer to which the bitmapped characters 
have been transferred. 

A problem associated with the use of portable scan- 
10 ners is that the scanner has to know the distance between 
the scanned "slices" in order to be able to identify the 
characters correctly. This means that the scanner either 
has to be passed over the characters at a constant speed, 
which is very difficult to achieve manually, or else con- 
15 tain a very accurate distance meter. 

In the above-mentioned US 5,301,243, the distance 
problem is obviated with the aid of a mechanical distance 
meter in the form of a wheel. However, a mechanical dis- 
tance meter comprises moving parts, which is a drawback 
20 as far as durability and manufacturing are concerned. 

Moreover, the distance meter makes it difficult to record 
information from a newspaper or the like if the latter is 
not placed on a hard surface. In addition, the distance 
meter impairs the possibility of seeing the information 
25 which is to be recorded, since the wheel has to abut 

against the surface from which the information is to be 
recorded and consequently makes the end of the scanner 
which abuts against the surface bulkier. 

A further problem connected with known portable 
30 scanners is that they must be held orientated substan- 
tially in one and the same predetermined direction the 
whole time. More specifically, the sensor must be orien- 
tated perpendicular to the direction in which the pen 
is being moved, i.e. perpendicular to the character 
35 sequence. If a scanner is held in another direction or it 
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rotated around its longitudinal axis while a sequence of 
characters is being recorded, the scanner may have diffi- 
culty identifying the characters recorded, since there is 
no directional information in the scanned "slices". This 
5 is a serious drawback of the prior art since different 
individuals often hold a scanner in different ways. In 
particular, there is a difference in the way right-handed 
and left-handed individuals hold scanners. Furthermore, 
an individual often changes the position of his hands as 

10 the scanner moves along a line. This is a particular 

problem when one is holding a book or the like in one's 
hand and wishes to record text. 

US 4,949,391 discloses an optical scanner with no 
distance meter. More specifically, this optical scanner 

15 comprises a two-dimensional sensor, which records images 
with partially overlapping contents of a text over which 
the scanner is passed. However, the handheld scanner is 
really only an "unintelligent" recording unit which 
records characters without interpreting them. Instead, 

20 the OCR processing is carried out in a computer to which 
the scanner is connected. The scanner correlates with 
each other images recorded one after another in order to 
remove columns with redundant information from the images 
so that they will require less storage space. The scanner 

25 can only be moved in a predetermined direction which is 
defined by a wheel. 
Summary of the Invention 

One object of the invention is to obviate, or at 
least reduce, the problems described above of the prior 

30 art for portable scanners, and specifically to provide 
a method and a pen which can be used for recording a 
selected sequence of characters without the aid of a dis- 
tance meter and with no requirement of a constant speed 
of movement. 
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In this connection, a further object is to provide 
such a method and such a pen which can be held rotated 
in different ways around its longitudinal axis and which 
specifically allows the pen to be rotated around its lon- 
5 gitudinal axis while a selected sequence of characters is 
being recorded, without the reliability of the identifi- 
cation of recorded characters being reduced in a decisive 
manner. 

The objects are achieved by a pen and a method for 
10 recording a selected character sequence which have the 
features stated in claims 1 and 1, respectively. 

As mentioned above, the pen and the method accord- 
ing to the invention are used for recording a selected 
sequence of characters by passing the pen over the char- 
15 acter sequence. The sequence selected can, for example, 
consist of one or several words or one or several sen- 
tences selected from a longer text. The characters can 
be letters, numbers and or other written characters or 
symbols. The characters can be scanned from a paper or 
20 another medium, which can be scanned optically. 

The pen comprises a light-sensitive sensor with a 
two-dimensional sensor surface. In this context, a two- 
dimensional sensor surface refers to a sensor surface 
which can record more than one pixel in two directions 
25 perpendicular to one another. The sensor is preferably 

of the CCD type. It can record two-dimensional images in 
grey scale or in colour. The images consist of a plura- 
lity of pixels, each being defined by an intensity value. 
The height of the images should be such that there is 
30 essentially room in the image for the highest character 
which is to be recorded. The width of the images can be 
chosen on the basis of the image-recording frequency 
desired and available sensors. 

Furthermore, the pen comprises a signal processing 
35 unit which should contain at least a central processing 
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unit, a memory, input/output circuits, and program in- 
structions for implementation of the method according to 
the invention. The signal processing unit preferably com- 
prises a suitably programmed microprocessor. The signaX^\ 
5 processing unit employs the partially overlapping con- 
tents of the images to put the recorded images together 
into one image. The putting-together is done both verti- 
cally and horizontally so that the vertical as well as 
the horizontal directional information in the images is 

10 utilised. As a result, there is no duplication of the 
contents in the composite image. 

When an image has been put together, the signal 
processing unit identifies the characters in the image 
and stores them in character-coded format in a memory in 

15 the pen. This means that the pen is of the stand-alone 

type. However, it is suitably connectable to an external 
computer, so that the information can be transferred to 
the latter. A composite image preferably has a predeter- 
mined maximum length- In order to increase the speed and 

20 reduce the memory requirement for images which have been 
recorded but not yet put together, the putting-together 
is suitably carried out sequentially as the images are 
being recorded. Thus, the fact that the putting-together 
can be performed in real time reduces the memory require- 

25 ment for recorded images. The putting-together normally 
continues until all recorded images have been put toge- 
ther. However, it is interrupted if the predetermined 
maximum length of the composite image is reached. Alter- 
natively, the putting-together can be carried out after 

30 the recording is complete, in which case a predetermined 
maximum number of images can be recorded, which are sub- 
sequently put together into an image. The pen can have 
an indicating device for indicating that it is ready to 
resume recording. 
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By using a sensor with a two-dimensional sensor sur- 
face, by means of which images with partially overlapping 
contents can be recorded, and by putting together the 
images into one image, it is possible to eliminate the 
5 requirement of a constant movement speed or of the provi- 
sion of a distance meter. By virtue of the contents of 
the images partially overlapping, the images will them- 
selves contain information about their relative position, 
i.e. distance information, and as a result there is no 
10 need to record this information using a distance meter. 
Moreover, the user can hold the pen at an optional angle 
of rotation in relation to the text and can even rotate 
the pen while recording the images without impairing the 
character identification, since the overlapping images 
15 contain directional information, which indicates their 
relative position. This considerably improves user- 
friendliness - 

Preferably, the pen has a display, so that the user 
can read the recorded character sequence without first 
20 having to transfer it to an ordinary computer. 

As mentioned above, in order for it to be possible 
to carry out the putting-together, there has to be a cer- 
tain overlapping of the contents of the recorded images. 
The degree of overlapping should preferably be greater 
25 than 20%, preferably greater than 50%. 

To make the contrast in the recorded images suffi- 
ciently great and to avoid blurriness caused by the pen 
being moved in relation to the character sequence, the 
characters which are to be recorded are preferably illu- 
30 minated by light pulses when the pen is being passed over 
the character sequence. The illumination can suitably be 
achieved with the aid of illumination means, e.g. an LED, 
which is fitted in or to the apparatus. The frequency of 
the light pulses should be the same as and be synchro- 
35 nised with the frequency for recording images, i.e. the 
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frequency for reading the contents of the light-sensitive 
sensor. A fast mechanical shutter, for example, could be 
used instead of the pulsed illumination. However, this 
results in a more complicated apparatus. 
5 When the images are being put together, images 

recorded one after another are preferably examined in 
pairs with respect to possible overlap positions for 
determining the overlap position which affords the best 
correspondence between the contents of each pair. The 

10 overlap position determined in this manner is used v/hen 
putting together the images of the pair. 

When the best overlap position has been determined, 
a pixel intensity value is preferably determined for each 
overlapping pixel. This pixel intensity value is based 

15 on the intensity values of the pixel in question in the 

images of the pair. The pixel intensity values determined 
in this manner are used in the final composite image. 

When the whole image has been put together, the 
image is preferably divided into a plurality of sub- 

20 images each comprising a character. This division, which 
thus does not have to be a physical division but rather 
can be performed in the whole image, constitutes a prepa- 
ration for the actual character recognition, each charac- 
ter preferably being interpreted separately. Alternative- 

25 ly, whole words or parts thereof could be interpreted. 

The division of the image into sub-images can be 
effected by the intensity of the pixels being added up 
row by row and column by column in the composite image in 
order to obtain a plurality of row sums and a plurality 

30 of column sums and by local minima being identified among 
the row sums and column sums obtained. 

The identification of the characters in the compo- 
site image is advantageously effected with the aid of a 
neural network. 
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Suitably, in this connection, the intensity value of 
each pixel in a sub-image is determined and is fed as an 
input signal to the neural network, which is adapted to 
identify the character in said sub-image with the aid of 
5 the input signals. 

Prior to the identification of the characters, the 
composite image can be rotated if necessary so that the 
characters in the image form a horizontal row. This faci- 
litates the identification. 
10 The pen according to the invention can for example 

be employed to record text while studying, to scan nume- 
rical values, to compose new documents on the basis of 
old documents which exist in paper form only, and for 
many other applications , 
15 Brief Description of the Drawings 

The present invention will be described below by way 
of an example which illustrates how the invention can be 
" realised and used. The description refers to the accom- 
panying drawings, in which 
20 Fig, 1 schematically shows an embodiment of a high- 

lighter pen according to the invention; 

Fig. 2 is a block diagram of the electronic cir- 
cuitry in an embodiment of the highlighter pen according 
to the invention; 
25 Fig. 3 is a flowchart and shows an implementation of 

the method according to the invention; 

Figs 4a-4e illustrate an example of an implementa- 
tion of the method according to the invention; 

Figs 5a-5c illustrate another example of an imple- 
30 mentation of the method according to the invention; and 

Figs 6a-6c illustrate a further example of an imple- 
mentation of the method according to the invention. 
Description of a Preferred Embodiment 

In the embodiment of the pen according to the inven- 
35 tion shown in Fig. 1, it has a casing 1 with approximate- 
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ly the same shape as a conventional highlighter pen. One 
short end of the casing 1 has a window 2, which is in- 
tended to abut against or be held a short distance from 
a storage medium, usually a sheet of paper, from which 
5 characters are to be recorded. The width of the window 2 
is chosen on the basis of the desired image-recording 
frequency, the desired maximum traction speed and avail- 
able sensors. The height of the window 2 is somewhat 
greater that the highest character which is to be record- 

10 ed. The window 2 forms an acute angle with the longitudi- 
nal axis of the pen, so that the user is guided to hold 
the pen at a predetermined angle to the sheet of paper. 
Moreover, the window is somewhat retracted into the cas- 
ing so that it will not wear against the paper . 

15 The casing 1 essentially contains an optical part 3, 

an electronic part 4 and a power supply 5. 

The optical part 3 comprises an LED 6, a lens system 
7 and a light-sensitive sensor 8 which constitutes an 
interface to the electronic part 4. 

20 The LED 6 has the task of illuminating the part of 

the storage medium which is currently under the window. 
A diffuser 9 is mounted in front of the LED 6 to diffuse 
the light. 

The task of the lens system 7 is to project an image 
25 of the part of the storage medium which is located under 
the window 2 of the light-sensitive sensor 8 as accurate- 
ly as possible. If the sensor 8 has a smaller light-sen- 
sitive area than the window 2, the lens system 7 must 
also reduce the image. The skilled person will be able to 
30 construct a number of different lens systems to fulfil 
this purpose. 

In this example, the light-sensitive sensor 8 con- 
sists of a- two-dimensional square CCD unit (CCD =^ charge 
coupled device) with a built-in A/D converter. Such sen- 
35 sors are commercially available. Here, the sensor 8 is 
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mounted at a small angle to the window 2 and on its own 
printed circuit board 11. 

The power supply to the pen is obtained from a bat- 
tery 12 which is mounted in a separate compartment 13 in 
5 the casing. 

The block diagram in Fig. 2 schematically shows the 
electronic part 4. It comprises a processor 20, which by 
the intermediary of a bus 21 is connected to a ROM 22, in 
which the programs of the processor are stored, to a 

10 read/write memory 23, which constitutes the working memo- 
ry of the processor and in which the images from the sen- 
sor as well as the identified and interpreted characters 
are stored, to a control logic unit 24, as well as to the 
sensor 8 and to the LED 6. The control logic unit 24 in 

15 turn is connected to a number of peripheral units, which 
comprise a display 2, which is mounted in the casing, an 
IR transceiver 26 for transferring information to/from an 
external computer, buttons 27, by means of which the user 
can control the apparatus, a tracer LED 28 which emits a 

20 light beam which makes it easier for the user to follow 

the text, and an indicating device 29, e.g. an LED, indi- 
cating that the pen is ready to record characters. Con- 
trol signals to the memories, the sensor, and the peri- 
pheral units are generated in the control logic unit 24. 

25 The control logic unit also handles generation and prio- 
ritisation of interruptions to the processor. The buttons 
27, the IR transceiver 26, the display 25 and the tracer 
LED 28, and the LED 6 are accessed by the processor writ- 
ing and reading in the register in the control logic 

30 unit. The buttons 27 generate interruptions to the pro- 
cessor 20 when they are activated. 

The pen works as follows. A user holds the pen 
against a sheet of paper with text in the location where 
he wishes to start recording a character sequence. He 

35 pushes a button 27 to activate the pen and passes the 
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latter over the text he wishes to record, following the 
text in the same way as when one reads a text. The tracer 
LED 28 emits a light beam which facilitates following the 
lines. When the user activates the pen, the processor 20 
5 commands the LED 6 to begin generating strobe pulses at a 
predetermined frequency of 25 Hz, the images recorded by 
the sensor being stored in the read/write memory 23 at 
the. same frequency. However, text which is stored in the 
form of images requires a large amount of memory space 

10 and cannot immediately be used by a computer in, for 
example, a word-processing program. Consequently, the 
characters in the images are identified and stored with 
the aid of ASCII code. When the user has passed the pen 
over a selected sequence of characters or has come to 

15 the end of a line, he lifts the pen from the paper and 
releases the activating button, whereby the processor 
20 turns off the LED 6. When the selected character 
sequences have been recorded, the user can control the 
pen to show recorded text on the display 25 or to trans- 

20 fer the text to an external computer by the intermediary 
of the IR transceiver 12. The possibility of showing the 
scanned character information directly on the display has 
proven very essential since a user often wishes to verify 
that the right character has been scanned. 

25 The flowchart in Fig. 3 shows an implementation of 

the method according to the invention. In step 301, 
images with overlapping contents are scanned with the 
aid of the pen described above and are stored in a cur- 
rent image area in the read/write memory 23. The images 

30 are stored as images, i.e. with the aid of a plurality of 
pixels, each having a grey scale value in a range from 
white to black. 

As soon as a whole picture is stored in the current 
image area, the putting-together of the image with the 

35 previous image is suitably commenced, step 302, if such 
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an image is available. If there is no previous image, the 
current image is entered directly into a line image area 
in the read/write memory - 

In order to determine how the current image is to 
5 be put together with the previous image so that the best 
correspondence is achieved between the contents of the 
images, every possible overlap position between the 
images is examined, at the pixel level, and an overlap 
measurement is determined as follows: 
10 1) For each overlapping pixel position, the grey 

scale values of the two pixels included are added up if 
the latter are not white. A pixel position in which none 
of the pixels are white is designated a plus position. 

2) The grey scale sums for all the plus positions 

15 are added up. 

3) The neighbours of each pixel position are exa- 
mined. If an overlapping pixel position is not a neigh- 
bour of a plus position and consists of a pixel which is 
white and a pixel position which is not white, the grey 

20 scale value of the non-white pixel is subtracted, pos- 
sibly multiplied by a constant, from the sum in point 2) . 

4) The overlap position which provides the highest 
overlap measurement as stated above is selected. In the 
resulting composite image the mean value of the grey 

25 scale value of the overlapping pixels is used. In this 
way, noise can be suppressed in the overlap area. The 
putting-together is thus carried out both vertically and 
horizontally. If it is detected that, when being put 
together, the images do not end up on a horizontal line, 

30 the composite image is suitably adjusted so that it 

becomes horizontal, for example by rotating the composite 
image . 

The composite image gradually develops in the line 
image area in the read/write memory. It is preferred that 



wo 98/20446 



PCT/SE97/01827 



the line image area be big enough to store an A4 line of 
normal typewritten text. 

In step 303, the software of the processor 20 
divides the composite image in the line image memory area 
5 into sub-images each containing only one character. The 
purpose of this is to create input signals to a neural 
network software which is to interpret the characters. 
The division is effected by adding up the grey scale 
values of the pixels for each pixel row and each pixel 

10 column in the composite image. By studying the local 
intensity minima for the row sums and column sums ob- 
tained, boundaries for the extent of each character in 
the image can be determined. 

If the composite image contains more than one line 

15 of text, which may be the case if the text is small, the 
lines of text must be separated and the line in question 
must be selected before the division into sub-images is 
carried out. The separation can be performed using a 
technique like the one used for the division into sub- 

20 images. 

In step 304, each of the sub-images is scaled down 
to a predetermined pixel format by the sub-image being 
divided into groups of pixels, each being replaced by 
a pixel whose grey scale value corresponds to the mean 

25 value of the grey scale values of the pixels included in 
the group. If required, a like downscaling can be carried 
out between other steps in the method according to the 
invention. Further, the character is centred with respect 
to its point of balance and the grey scale values are 

30 normalised so that the sum of the square of the grey 
scale value of each pixel is given a fixed value. 

Subsequently, in step 305, each character in the 
composite image of the character sequence depicted is 
interpreted. The grey scale values of the pixels which 

35 together constitute a sub-image containing only one 
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character are fed as input signals to a neural network. 
Each output from the neural network represents a charac- 
ter which the network can identify- The output from the 
network which has the highest output signal is chosen and 
5 the character thus chosen is stored in step 306 using a 
predetermined character code format, for example ASCII 
code, in the read/write memory 23 in a memory area for 
interpreted characters. When the character identification 
and storing in character-coded format is completed, the 
10 processor activates the indicating device 29 to inform 
the user that it is ready to record a new character 
sequence, step 307. Subsequently, it goes back to step 
301. 

The steps described above are thus carried out by 

15 the processor 20 with the aid of the associated units 
and suitable software. Such software can be created by 
the skilled person with the aid of the above instruc- 
tions. The character recognition is performed with the 
aid of neural network software, which is trained in a 

20 suitable manner. Neural network software is commercially 
available from, for example, MATLAB Neural Network 
Toolbox, The MathWorks Inc., 24 Prime Park Way, Natick, 
MA 01760, USA. 

Figs 4a-4e illustrate how the method according to 

25 the invention works when the character sequence "Flygande 
backasiner" is scanned. Fig. 4a shows the text on a sheet 
of paper, which also has a certain amount of "dot noise" 
in the form of small black spots. Fig. 4b shows the 
images which are being recorded with the aid of the sen- 

30 sor. As can be seen from this figure, the contents of the 
images partially overlap. The letter 1 appears completely 
in image No. 1 and partially in image No. 2. The degree 
of overlapping depends on the traction speed, i.e. the 
speed with which the user pulls the apparatus over the 

35 text in relation to the frequency with which the contents 
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of the sensor are read out. Fig. 4c shows what the whole 
composite image looks like. Note that the image is still 
stored in the form of pixels. Fig, 4d illustrates the 
division of the whole image into sub-images. Fig. 4e 
5 shows the scaled and normalised letters which are used as 
input signals to the neural network. When the method has 
been implemented the text "Flygande backasiner" is stored 
in the read/write memory of the pen as ACSII code. 

In the example shown, the pen was held straight and 

10 was not rotated at all while the character sequence was 
being recorded. If the pen had been held rotated at a 
constant angle around its longitudinal axis, the images 
would instead have looked like the ones in Fig. 5a. In 
this case, when the images are put together a slanting 

15 image is obtained, see Fig. 5b, which has to be rotated 
in order for the character sequence to be horizontal. 
Fig. 5c. In order to determine how great a rotation is 
to be performed, the position of the characters in the 
composite image is detected in a couple of places. 

20 If the pen is rotated around its longitudinal axis 

at the time of recording, the inclination of the images 
will be different. An example of this is shown in 
Fig. 6a. When the images are put together, see Fig. 6b, 
a wave-shaped character sequence is obtained. It can be 

25 straightened out to a horizontal line by the position of 
the characters in the composite image being detected in a 
plurality of locations and by different segments being 
rotated to different extents (see Fig. 6c). 
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CLAIMS 

1. A pen for recording a selected sequence of char- 
5 acters by passing the pen over the sequence of charac- 
ters, comprising a light-sensitive sensor (8) for imag- 
ing the characters and a signal processing unit (20-24) 
for identifying the imaged characters and storing them 
in character coded format in the pen, charac'ter- 

10 i s e d in that the light-sensitive sensor (8) has a two- 
dimensional sensor surface, the sensor being adapted to 
image the selected character sequence with the aid of a 
plurality of images with partially overlapping contents, 
and that the signal processing unit (20-24) is adapted to 

15 put together said plurality of images both vertically and 
horizontally into a composite image, whereby compensation 
is obtained for different speeds of movement and diffe- 
rent angles of rotation of the pen, and that the identi- 
fication of the imaged characters is performed on the 

20 basis of the composite image. 

2. A pen according to claim 1, further comprising 
illumination means (6), which are adapted to emit light 
pulses for illuminating characters which are to be 
recorded. 

25 3, A pen according to any one of the preceding 

claims, wherein the signal processing unit (20-24) com- 
prises a neural network for identification of the char- 
acters - 

4. A pen according to any one of the preceding 
30 claims, wherein the signal processing unit is adapted 

to divide the composite image into a plurality of images 
each comprising only one character. 

5. A pen according to any one of the preceding 
claims, further comprising an indicating device (29) for 
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indicating that the pen is ready to scan the character 
sequence . 

6. A pen according to any one of the preceding 
claims, further comprising a display (25) for showing 

5 the identified characters. 

7 . A method for recording a selected sequence of 
characters with the aid of a pen, comprising the steps of 

passing the pen over the selected character sequence 
and simultaneously imaging the selected character se- 
10 quence with the aid of a plurality of images, whose con- 
tents partially overlap, 

putting together said plurality of images both ver- 
tically and horizontally into a composite image with the 
aid of a signal processing unit in the pen, whereby com- 
15 pensation is obtained for different speeds of movement 
and different angles of rotation of the pen, 

identifying the characters in the composite image, 

and 

storing the identified characters in the pen with 
20 the aid of a predetermined character code format. 

8. A method according to claim 7, further comprising 
the step of illuminating the characters which are to be 
recorded with light pulses, when the pen is being passed 
over them. 

25 9. A method according to claim 7 or claim 8, wherein 

the step of putting together said plurality of images 
comprises the examination in pairs of images recorded one 
after another with respect to possible overlap positions 
for determining the overlap position which affords the 

30 best correspondence between the contents of each pair of 
images recorded one after another, the overlap position 
determined in this manner being used in putting together 
the images of the pair. 
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10. A method according to any one of claims 7-9, 
wherein the putting-together of the recorded images is 
carried out in real time. 

11. A method according to claim 9, wherein the 

5 recorded images consist of a plurality of pixels each 
being defined by an intensity value, further comprising 
the steps of 

when putting together the images of the pair after 
determining the best overlap position, determining a 
10 pixel intensity value, preferably a mean value, for each 
overlapping pixel, which pixel intensity value is based 
on the intensity values of the pixel in question in the 
images of the pair, and 

using the pixel intensity values determined in this 
15 manner in the composite image. 

12. A method according to any one of claims 7-11, 
wherein the step of identifying the characters comprises 
dividing the composite image into a plurality of sub- 
images each comprising only one character. 

20 13. A method according to claim 12, wherein the com- 

posite image consists of a plurality of pixels, each 
being defined by an intensity value, and wherein the step 
of dividing the composite image into a plurality of sub- 
images comprises adding up the intensity of the pixels 
25 row by row and column by column in the composite image 

for obtaining a plurality of row sums and column sums as 
well as identifying local minima among the row sums and 
column sums obtained for dividing the composite image 
into sub-images. 

30 14. A method according to claim 12 or 13, wherein 

each sub-image consists of a plurality of pixels, each 
being defined by an intensity value, and wherein the step 
of identifying the characters in the recorded images com- 
prises determining the intensity value of each pixel in a 

35 sub-image and feeding it as an input signal to a neural 
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network, which is adapted to identify the character in 
said sub-image with the aid of the input signals, 

15. A method according to any one of claims 7-14, 
further comprising the step of, prior to the identifica 
5 tion of the characters, rotating the composite image so 
that the characters in said image form a horizontal row 
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